The loss in two hours was a bitter blow for the Briton who has been a runner-up here three times while Ancic goes on to face Federer in the semi-finals on Saturday.
Henman lost the opening set after 58 minutes after four breaks of serve had been exchanged and the British number one had saved a pair of set points.
The second was more of the same, with the Briton losing a 5-3 lead.
In the breaker, Henman fell to 0-3, recovered but couldn't get on top of his rival. A running forehand wide from the Briton gave Ancic a pair of match points and he ended it moments later.
Swiss top seed Federer, aiming to bounce back from his disappointing Australian Open semi-final exit, had to shake off two break points from the 15th-ranked Russian Davydenko in the penultimate game of the match before advancing. "He made me work very hard, it was a close match," said Federer, now 13-1 in the season with a title at the start of the year in Doha.
The win in one hour and 41 minutes was Federer's fifth without loss against Davydenko.
Croatia's Ivan Ljubicic booked his place in the semi-finals but only after an angry 3-6, 6-4, 6-2 victory over Thailand's Paradorn Srichaphan.
Ljubicic said the Asian number one broke into a "victory dance" after earning a break point at four-all in the second set of their quarter-final clash.
The gesture, innocent as it was, fired the Bosnian-born former war refugee into action.
Meanwhile Thomas Johansson of Sweden clinched a 6-2, 7-6 (7/5) quarter-final victory over Radek Stepanek of the Czech Republic.
The 1998 losing finalist here said that his win left him even more satisfied than Thursday's upset of huge-hitting young compatriot Joachim Johansson.
COLLATED (x denotes seed):
(QUARTER-FINALS): Roger Federer (SUI x1) bt Nikolay Davydenko (RUS x6) 7-5, 7-5; Thomas Johansson (SWE) bt Radek Stepanek (CZE) 6-2, 7-6 (7/5); Ivan Ljubicic (CRO) bt Paradorn Srichaphan (THA) 3-6, 6-4, 6-2; Mario Ancic (CRO) bt Tim Henman (GBR x3) 7-6 (7/5), 7-6 (7/4).